Discovering Cyclic Causal Models in Psychological Research

Network Theory in Psychology

Mental disorder is produced by direct causal interactions between symptoms that reinforce each other via feedback loops. (Borsboom & Cramer, 2013).

Inferring causality from observational data

And what has happened a lot in psychology is that many empirical researchers try to gain insights into these causal relationships by fitting this statistical network model using observational data. [Alt + click!] But these connections in the network are merely statistical relations, which cannot be directly translated into causal relations.

And more and more people have become aware of this issue, and there has come another popular casual modeling tool, which is DAG. DAG stands for directed acyclic graphical model. It has all these arrows you can interpret as a cause and effect. [Alt + click!] It seems very promising, however, one of biggest shortcomings of DAG is by definition it does not allow cycles. But as we saw before in the previous slide, the true system of psychopathology we are interested in, contains cycles. And in fact, the cycles are critical in understanding this central dynamic of symptoms. In that sense, we cannot really use DAG to estimate the underlying mechanism. So what do we need to do?

Directed Acyclic Graph (DAG)

Directed Cyclic Graph (DCG)

Problem: Estimating a cyclic causal model is fundamentally very difficult.

Solutions exist

Cyclic Causal Discovery (CCD) (Richardson, 1996)
Fast Causal Inference (FCI) (M. Mooij & Claassen, 2020)
Cyclic Causal Inference (CCI) (Strobl, 2019)

This is the part of the results. I used four different evaluation metrics, SHD, precision, recall, and uncertainty rate. But here I will present the SHD which stands for structural hamming distance. It sounds horrible but it is just counting all the steps going from the estimated graph to the true graph. So the lower SHD, the better, meaning that the estimated graph is closer to the true graph, hence taking less steps.

Here we see that overall CCI the green line and FCI the yellow line perform better in sparse conditions while CCD the red line perform better in dense conditions.

When we see the rows, p refers to the number of variables so I have 5-variable model and 10-variable models. Between small and big models, we see that in small models the performance difference between algorithm is more obvious compared to the big models.

And we did not observe any significant difference between the conditions involving latent confounders, LC, here and without the latent confounders.

So in general, it is not so straightforward which algorithm is the best. It is somewhat dependent on the situations and characteristics of the causal structure.

Summary

Causal inference is the fundamental interest in science.
The underlying dynamic processes of many systems contain cycles.
Our study showcases the cyclic causal discovery algorithms that are potentially suitable for typical psychological observational data.
Conclusion is rather nuanced (no one-size-fits-all algorithm).
Causal discovery methods could provide much richer insights into the underlying causal dynamics of the system than statistical network models.

To sum it up, causal inference is the fundamental interest in science regardless of field of study.

And often, the underlying causal mechanism of interest contains cycles.

In this project, we show that there are methods available that can estimate the cyclic causal models using observational data.

But, given our simulation study, our conclusion is rather nuanced. We found that there is no single algorithm that performs well across all scenarios. So, we believe that researchersmust consider the characteristics of the causal system of interest when choosing an algorithm.

Regardless, we emphasize that these causal discovery methods are more informative than statistical network analysis when exploring causal structures. And I do hope that this current project encourages researchers to adopt these methods and study the potential applicability in psychology.

References

Borsboom, D., & Cramer, A. O. J. (2013). Network analysis: An integrative approach to the structure of psychopathology. Annual Review of Clinical Psychology, 9(1), 91–121. https://doi.org/10.1146/annurev-clinpsy-050212-185608

Eberhardt, F., Hoyer, P., & Scheines, R. (2010). Combining experiments to discover linear cyclic models with latent variables. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 185–192.

M. Mooij, J., & Claassen, T. (2020). Constraint-based causal discovery using partial ancestral graphs in the presence of cycles. In J. Peters & D. Sontag (Eds.), Proceedings of the 36th conference on uncertainty in artificial intelligence (UAI) (Vol. 124, pp. 1159–1168). PMLR. https://proceedings.mlr.press/v124/m-mooij20a.html

Richardson, T. (1996). Discovering cyclic causal structure. Carnegie Mellon [Department of Philosophy].

Spirtes, P. (1994). Conditional independence in directed cyclic graphical models for feedback. Carnegie Mellon [Department of Philosophy].

Strobl, E. V. (2019). A constraint-based algorithm for causal discovery with cycles, latent variables and selection bias. International Journal of Data Science and Analytics, 8(1), 33–56. https://doi.org/10.1007/s41060-018-0158-2

Theoretical complication

Global Markov property is no longer guaranteed.
- Need extra restrictions on \(P\) (Spirtes, 1994).
  E.g., linear relations with independent error terms (\(\varepsilon\)).
Not all cyclic models are statistically identified.
- Need additional constraints on \(B\): It needs to converge to an equilibrium (All \(|\lambda| < 1\))(Eberhardt et al., 2010).
Output of algorithm is PAG (partial ancestral graph).
- Large Markov equivalence class.

There are many theoretical challenges when estimating causal model with cycles. But I picked three here, the most crucial ones in my opinion. So when it comes to cyclic models, the most crucial issues are first, the global markov property which is the necessary condition in order to read off the causal relationships from the graphical models, that is often violated in the models with cycles. Spirtes proved that under the additional assumptions, such as linear relationships with independent error terms, the global markov property holds even in cyclic models, but that is pretty strict assumption you have to claim.
And another major issue is that not all cyclic models are statistically identified, meaning that the structural equations defining the model can have either too many solutions possible or no solution at all. Again, we need additional constraints on B, the weight matrix. When the eigen values of B are all smaller than one in absolute value, then we can ensure that there is a solution such that the cyclic model reaches an equilibrium inseted of diverging to infinite.
Lastly, the algorithm typically cannot identify direct causal relationships in cyclic models, but recovers only upto ancestral causal relationships. This is again due to the complexity arising from allowing cycles and this leads to a larger Markov equivalence class of graphs, meaning that there are many more statistically equivalent graphs.

So this causes quite some headaches when trying to estimate the cyclic models.

Solutions exist, but…

Project goals

Give an accessible overview of the algorithms.
Investigate the performance of each algorithm.
Apply to empirical data to assess practical applicability.

Simulation settings

So this is the simulation settings. We vary the number of edges, so the density of the model, as you see in the columns, and the size of the models, so the number variables, you see that in the rows Also, we differ the conditions by adding the latent confounder, which is represented as L1 and L2 in the figure. Lastly, we also vary the sample sizes ranging from 50 to 10,000. In the end, we have 2 different densities, 2 different model sizes, and either absence or presence of confounder, and 10 different samples. So that lead us to 2 by 2 by 2 by 10 simulation design. And for each of this different condition, we run all three algorithms 500 times, So that is 80 conditions times by 3 algorithms times by 500 iterations. So in the end we estimate 120,000 different graphs in our simulation. not really important number just give you a idea.

Constraint-based algorithm output

CPDAG
CCD PAG
FCI PAG
CCI PAG

Performance evaluation

So what do we then compare or evaluate the algorithms? The basic idea is that, we have this true DCG, so directed cyclic graph. And we construct the true ancestral graph corresponding to this underlying DCG. Why do we do that… Well, it’s because the output of these algorithms are called PAG that I briefly described earlier. So these cyclic causal discovery algorithms fundamentally cannot estimate the true underlying directed cyclic graph, but they aim to estimate this ancestral graph. This is definitely due to the challenges coming from the cyclicity nature. We can delve deeper into this if anyone is curious. But for now, remember that we are here aiming to recover the true ancestral graph.

So then, basically we assess the performance of each algorithm by comparing their outputs, the PAGs to the true ancestral graph. so how well they can retrieve the true model?

Evaluation metrics

Structural Hamming distance (SHD) = \(A\) (Addition) + \(D\) (Deletion) + \(C\) (Changes)
Precision = \(\frac{TP} {(TP + FP)}\)
Recall = \(\frac{TP} {(TP + FN)}\)
Uncertainty rate = \(\frac{\text{Number of circle endpoints} (\circ)}{\text{Total number of possible endpoints}}\)

Example:
- For the arrow head (\(>\)): precision = \(\frac{4}{4 + 3 + 0}\) and recall = \(\frac{4}{4 + 0 + 0}\).
- The value of SHD for the example PAG output from (b) – provided that the true ancestral graph is (a) – is 6: 0 (A) + 0 (D) + 6 (C).

Well, how do we compare exactly? So these are evaluation metrics we use in our simulation. We first look at the structural hamming distance as an overall performance. So this more of a global evaluation metric. And we also look at three different local metrics, which are precision, recall, and uncertainty rate. They are defined as such, which is not immediately intuitive.

But these are actually very simple. So this is a simple example. Let’s say (a) is the true ancestral graph. and (b) is the estimated graph. Then first, we can compute the structural hamming distance, the name sounds a bit scary but it is just a simply summing all the steps we need going from one graph to the other. So in this case, how many steps do we need from the estimated graph (b) to the true graph (a): first we see that there is no edge needed to be added nor deleted. Then we see, we need to change six edge endpoints. therefore, the structural hamming distance in this case is six. Then moving on to the precision, recall, and uncertainty rate. This is the confusion matrix for this example; we have three different categories: the arrow head, arrow tail, and NULL which means there is no edge at all. One thing to note here is that we do not count the circle endpoint when computing precision or recall! This is pretty important point for later. Anyways, we can for example, compute the precision and recall for the arrow head. just simply following definition, precision is 4 over 7 and recall is 4 over 4, so 1 in this case. Lastly, uncertainty rate is again very simple. We just compute the proportion of circle endpoints. So in this case we have 3 circles so that gives us 3 over 20! where does 20 come from? 20 is the total possible edge endpoint in this graph. computed by 5 choose 2 times by 2 which is 20. So this was a long explanation on how we evaluated the performance of the algorithm.

Typical behavior of algorithms

(a): True ancestral graph
(b): PAG estimated by CCD
(c): PAG estimated by FCI
(d): PAG estimated by CCI

So these are the estimated graphs from each algorithm for the most simple condition, 5p sparse case. (a) is the true graph. and (b) is estimated by CCD, (c) is estimated by FCI, and (d) is estimated by CCI.

What is typically happening with these algorithm is basically that CCD tends to produce more circle but the directions it predicts tend to be all correct. On the other hand, FCI barely outputs any circle but it rather guesses quickly. But the directions are not very accurate. Lastly, CCI produces some circles but the directions are also likely to be correct. So this is what usually happens. CCD tends to be more conservative, and FCI is just guessing all the time (and often not correctly), while CCI is less conservative than CCD but it mostly gets the directions correct.

So that was our general observation from the simulation study.

Empirical example

Then we look at what happens when we apply them to an empirical data. This was psychological data where they measured depression symptoms from about 400 people, cross-sectional data. So (a) is the standard statistical network model, often called Gaussian graphical model. and (b) is the PAG estimated by CCD. (c) is the PAG estimated by CCI. and (d) is the PAG estimated by FCI.

Well, in this case, we don’t really know what is the underlying true causal structure. But just by comparing the results, we could say that the structures themselves do not look so different. They all kinda show this three-clusters. But then causal graphs are much more sparse than statistical network model, which implies that it is probably not a good idea even to use network model as a guide to the causal skeleton.

Overall, this example highlights that the PAGs recovered by cyclic causal discovery algorithms provide more insights into this possible causal relationships that cannot be obtained from statistical network model alone.

Also, we saw that different algorithms appeared to agree on many features of the causal structure but of course they show some different inference on causal directions. Then of course your question could be, which one should i believe? well… that’s a question that I cannot just answer you. Which algorithm you should go for, depends on multiple factors like what assumptions do you have in your causal structure of interest, like do you believe that there are a lot of latent confounders or do you believe the causal relationships are linear… because all these algorithm make somewhat different assumptions as i showed earlier. So you need to think about that. And also, based on our simulation results, you can also consider what your priority is in terms of your research question. Do you care more about not getting wrong inferences or do you care more about getting the most inferences at the expense of getting some directions wrong, then you might prefer CCI. So it depends, and given the information that I’ve gotten so far. my advice, or my conclusion is it needs some more work, actually a lot of more work to do, and for now, there seems to be one single algorithm that performs well in every scenario.

Possible practical application

Personalized psychotherapy (target symptoms)
Medical: effective treatment design

Q: “what are the potential application of this project, so what does this mean to a clinical psychologist?”

For example, a clinical psychologist could use this to customize the therapy based on the found causal dynamics. Let’s imagine that this is a person’s real depression model with cycles, then we can see that insomnia is the central node that has a lot of connections causing other depression symptoms. So we can effectively treat this patient’s depression by targeting the sleeping problems.
I think this would work more or less the same way to other medical treatments. You can imagine that this is a cyclic causal model for some other disease, right. Then, we can also design a more effective treatment once we are able to identify which is the most influential cause of the disease. So yeah this would be one of the crucial clinical implications of this project.

Follow-up

Possible combination with different types of causal discovery algorithm. \(\rightarrow\) Hybrid!

CCD+ GES (greedy equivalence search)

Q:“What could be an interesting follow-up or extension you could do if more fundings become available?”

So I just told you that the algorithm usually cannot identify the one true graph but instead give you this whole set. And that’s the general limitations of the causal discovery algorithms, that they often don’t give you the full picture.

So let’s say this is the true model.And usually, the algorithms can only identify this up to a certain extent. So you see these edges are left undirected. Then I think we can use another algorithm, to orient the rest of edges.

For example, we could combine CCD with GES, greedy equivalence search algorithm, which utilizes different types of information to find an optimal model.

So like this, we can combine these two and get the full picture.

This extension is very plausible in my opinion and promising but it’s not known yet, which combination would work the best in practice. So this would be a great work to do if there is extra resources available in the future.

Thank you